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(54) Multiplexing apparatus and method, transmitting apparatus and method, and recording 
medium 



(57) A multiplexing apparatus and method, a trans- 
mitting apparatus and method, and a recording medium 
arranged to prevent a delay of one of multiplexing 
scheduling processing and multiplexing processing in 
accordance with a schedule obtained by the multiplex- 
ing scheduling processing from influencing the other of 
these processings. A scheduler calculates schedule 
information on the basis of access unit information from 
an encoder and stores the calculated schedule informa- 
tion by supplying it to an intermediate buffer. If some 



additional data is to be multiplexed, the scheduler also 
calculates the additional data and stores the calculated 
additional data by supplying it to an intermediate buffer. 
A bit stream multiplexer reads out schedule information 
from the schedule information intermediate buffer, 
selects data designated by the schedule information 
from an elementary stream from the encoder and from 
additional data stored in the additional data intermedi- 
ate buffer, and outputs the data as a multiplexed stream. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates to a multiplexing 
apparatus and method, a transmitting apparatus and 
method, and a recording medium. More particularly, the 
present invention relates to a multiplexing apparatus 
and method, a transmitting apparatus and method, and 
a recording medium suitable for use, for example, in a 
situation where a bit stream of a digital signal formed 
from an image (moving image) signal and an audio sig- 
nal is recorded on a recording medium such as a mag- 
neto-optical disk or a magnetic tape and is reproduced 
to be displayed on a display, or in a situation where such 
a bit stream is transmitted from a transmitting terminal 
to a receiving terminal over a transmission channel in a 
television conference system, a video telephone sys- 
tem, broadcasting apparatuses or the like, and is 
received at the receiving terminal to be displayed. 

2. Description of the Related Art 

Fig. 7 shows the configuration of an example of a 
conventional multiplexing/demultiplexing apparatus. 

At a transmitting terminal, a digital image signal 
(video data) is supplied to a video encoder 101, which 
compresses and encodes a bit stream of video data in 
accordance with a standard, e.g., IS013818-2 (so- 
called MPEG (Moving Picture Experts Group) 2 video 
standard), IS011172-2 (so-called MPEG1 video stand- 
ard) or the like, and thereafter outputs the compressed 
and encoded data. A digital audio signal (audio data) is 
supplied to an audio encoder 102, which compresses 
and encodes a bit stream of audio data in accordance 
with a standard, e.g., IS013818-3 (so-called MPEG2 
audio standard), IS011172-3 (so-called MPEG1 audio 
standard) or the like, and thereafter outputs the com- 
pressed and encoded data. Another encoder 103 is 
supplied with digital data (other data) other than video 
data and audio data. The encoder 103 encodes and 
outputs the other data. 

A bit stream compressed and encoded and thereaf- 
ter output will be referred to as "elementary stream" as 
occasion demands. 

Elementary streams of video data, audio data and 
other data respectively output from the video encoder 
101, the audio encoder 102 and the other encoder 103 
are supplied to a multiplexer 104. The multiplexer 104 
multiplexes the elementary streams of video data, audio 
data, other data, and additional data (additional infor- 
mation) necessary in a time division multiplexing man- 
ner at a receiving terminal. A stream obtained by this 
multiplexing (hereinafter referred to as "multiplexed 
stream" as occasion demands) is recorded on a prede- 
termined recording medium 105 or is transmitted 



through a predetermined transmission medium 106. 

On the other hand, at the receiving terminal, the 
multiplexed stream reproduced from the recording 
medium 105 or received through the transmission 

5 medium 106 is supplied to a demultiplexer 107. The 
demultiplexer 107 separates the multiplexed stream into 
elementary streams of video data, audio data and other 
data, and additional data by referring to the additional 
data in the multiplexed stream. The elementary streams 

10 of video data, audio data and other data are respec- 
tively supplied to a video decoder 108, an audio 
decoder 109 and another decoder 110 to be respec- 
tively decoded into video data, audio data and other 
data in certain formats. 

is The decoded video data is supplied to, for example, 
a display unit such as a cathode ray tube (CRT) or a liq- 
uid crystal display to be displayed. The decoded audio 
data is supplied to, for example, a speaker or the like to 
be output. The other data is supplied to, for example, a 

20 predetermined recording medium to be recorded. 

With respect to the above-described multiplexing of 
elementary streams at a transmitting terminal, a stand- 
ard model (system target decoder (STD)) correspond- 
ing to a decoder at a receiving terminal is supposed and 

25 time division multiplexing is performed so that decoding 
(decode processing) can be correctly performed in the 
STD. 

Fig. 8 shows the configuration of an example of an 
STD in accordance with IS013818-1 (so-called MPEG2 
30 system) and IS01 1 172-1 (so-called MPEG1 system). 

A multiplexed stream formed by time-division-multi- 
plexing elementary streams is input to the STD. The 
multiplexed stream contains time information (clock ref- 
erence) as additional data, and is input to the STD at a 
35 time corresponding to the time information. The multi- 
plexed stream also contains multiplexing rate informa- 
tion MUX_rate as additional data, and is input to the 
STD at a data rate corresponding to this multiplexing 
rate information. 
40 The multiplexed stream input to the STD is supplied 
to a switch 111. The switch 1 1 1 selects one of input ter- 
minals of a video buffer 1 12, audio buffers 113 1 to 1 13 N 
and another buffer 1 14 to enable video data, audio data 
(N-channel audio data) or other data to be supplied to 
45 and stored in the corresponding buffer (video buffer 
1 12, audio buffers 1 13! to 1 13 N ) or buffer 114). 

Each of the elementary streams of video data, 
audio data (N-channel audio data) and other data 
respectively stored in the video buffer 112, the audio 
so buffers 1 1 to 1 1 3 N and the other buffer 1 1 4 is instantly 
read out by a length corresponding to its predetermined 
decoded unit. 

Such a decoded unit is called an access unit. With 
respect to video data, one frame, for example, is pre- 
ss scribed as an access unit. With respect to audio data, 
one audio frame, for example, is prescribed as an 
access unit The length of one audio frame of audio data 
varies depending upon the sampling frequency at which 
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the audio data was sampled. 

The multiplexed stream also contains the decoding 
time of each access unit as additional data. Each of 
access units of video data, audio data and other data 
stored in the video buffer 112, the audio buffers 1 13! to s 
11 3 N and the other buffer 1 14 is instantly read out at the 
corresponding decoding time to be supplied to a 
decoder 1 1 5, decoders 1 1 6! to 1 1 6 N , or a decoder 1 1 7. 
Video data decoded by the decoder 115 is output to a 
CRT display unit or the like via a reorder buffer 1 1 8 and w 
a switch 119. 

Each access unit is read out according to the 
decoding time to ensure synchronization between the 
elementary streams. That is, for example, synchroniza- 
tion between images and corresponding sounds is is 
ensured. 

Each of access units into which the data is sepa- 
rated by the switch 1 11 is stored till a predetermined 
time (the above-mentioned decoding time) in the video 
buffer 1 1 2, the audio buffers 1 1 3j to 1 1 3 N or the other 20 
buffer 1 14 of the STD. These buffers will be referred to 
collectively as "STD buffer". Accordingly, on the trans- 
mitting side, it is necessary to perform scheduling of 
time division multiplexing in order to prevent an overflow 
in the STD buffer (an excess of data accumulated in the 25 
STD buffer over the capacity of the STD buffer) or an 
underflow (failure to store one complete access unit in 
the STD buffer before the decoding time). 

Fig. 9 shows the formats of multiplexed streams 
prescribed in IS01381 8-1 , for example. 30 

A multiplexed stream is obtained by time-division- 
multiplexing at least one elementary stream and is 
formed of a plurality of packets. That is, each of elemen- 
tary streams is separated into packets, and the time of 
being input to the decoder (clock reference), the decod- 35 
ing time at which each access unit is decoded (dis- 
played), etc., are added to the packets, thus forming a 
multiplexed stream. 

In IS013818-1, as shown in Fig. 9, a transport 
stream (Fig. 9(A)) and a program stream (Fig. 9(B)) are 40 
prescribed. A transport stream is a stream for use in 
transmission in an environment having such an influ- 
ence as to cause errors such as a bit error and cell loss. 
For example, such a stream is used in transmission 
channels of asynchronous transfer mode (ATM) net- 45 
works and digital broadcasting. A program stream is a 
stream for use in transmission in an environment with a 
reduced possibility of occurrence of errors. In particular, 
such a stream is used with a recording medium in the 
form of a disk or a tape. so 

As shown in Fig. 9(A), a transport stream is formed 
of transport stream packets of a fixed length in which 
elementary streams (of video data, audio data, and 
other data), etc., are set, and other packets in which are 
set an adaptation field containing a clock reference, ss 
etc., and additional information (additional information 
in a narrow sense) including, if the transport stream is a 
multi-program transport stream, program specific infor- 



mation (PSI) on selection from a plurality of programs, 
extraction of the desired packet and the method of 
decoding. 

Each packet has a transport stream header, and 
the packets in which elementary streams are set may 
have packet headers, each packet header containing a 
decoding time, information on the contents of the 
packet, etc. 

On the other hand, as shown in Fig. 9(B). a pro- 
gram stream is formed of a pack in which a certain 
number of packets of a variable length are combined. A 
packet header is attached to each packet. With respect 
to packets in which elementary streams are set, a 
decoding time is set in the corresponding packet 
header. 

Also, a packet header is attached to each pack. In 
the program stream, a clock reference is set in the pack 
header. 

The program stream further contains a packet in 
which additional data (additional information in a narrow 
sense) such as a program stream directory (PSD) or a 
program stream map (PSM) is set. 

The aforementioned additional data (additional 
information in a broad sense) represents data other 
than elementary streams in the multiplexed stream. 

Time division multiplexing for obtaining the above- 
described multiplexed stream is performed based on 
two processings described below. 

That is, it is necessary to perform, as the first 
processing, scheduling which comprises forming suita- 
ble additional data and determining data to be multi- 
plexed and the sizes of the multiplexed data in order to 
prevent an overflow and an underflow in an STD buffer 
of an STD supposed to exist at a receiving terminal and 
in order to ensure synchronization between elementary 
streams. (This kind of processing will be referred to as 
"multiplexing scheduling processing" as occasion 
demands.) 

Then, it is necessary to perform, as the second 
processing, multiplexing of the elementary streams and 
the additional data on the basis of information on the 
data and the sizes determined by the multiplexing 
scheduling (hereinafter referred to as "schedule infor- 
mation" as occasion demands). (This kind of processing 
will be referred to as "bit stream multiplexing process- 
ing" as occasion demands.) 

Fig. 10 shows the configuration of an example of a 
multiplexing apparatus which performs the above- 
described multiplexing scheduling processing and bit 
stream multiplexing processing in a time division man- 
ner. 

In this multiplexing apparatus, a scheduler which 
performs multiplexing scheduling processing and a bit 
stream multiplexer which performs bit stream multiplex- 
ing processing are not definitely separated from each 
other, and these two processings are performed by a 
scheduler and bit stream multiplexer 122. 

That is, encoders 12^ to 121 N encode video data 



3 



5 



EP0 868 087 A2 



6 



and audio data into elementary streams, which are sup- 
plied to the scheduler and bit stream multiplexer 122. 
The scheduler and bit stream multiplexer 122 alter- 
nately performs multiplexing scheduling processing and 
bit stream multiplexing processing by using the elemen- 
tary streams input thereto and by setting a schedule unit 
corresponding to, for example, a packet unit. 

The problem of alternately performing multiplexing 
scheduling processing and bit stream multiplexing 
processing as described above, i.e., in a time division 
manner is that, rf one of the two processings is delayed 
for some reason, the other processing is also delayed. 

For example, a situation such as that shown in Fig. 
11A may be considered where, when multiplexing 
scheduling processing S and bit stream multiplexing 
processing M are being performed alternately, a wait of 
outputting of the multiplexed stream occurs at a certain 
time and bit stream multiplexing processing M(n + 1) of 
the (n + 1)th packet is thereby delayed. In such a situa- 
tion, multiplexing scheduling processing S(n + 2) of the 
next (n + 2)th packet cannot be performed before bit 
stream multiplexing processing M(n + 1) is finished. 
Thus, a delay of bit stream multiplexing processing 
directly influences the next multiplexing scheduling 
processing. 

Another situation shown in Fig. 11B, for example, 
may be considered where multiplexing scheduling 
processing S(n + 1 of the (n + 1)th packet is interrupted 
to be delayed. Also in such a situation, multiplexing 
scheduling processing M(n + 1) of the (n + 1)th packet 
cannot be performed. 

As described above, a delay of one of the two 
processings directly influences the other processing, so 
that processing delay times are successively accumu- 
lated. When the length of the accumulated delay time is 
increased to a certain value, it becomes difficult to per- 
form time division multiplexing in a real time required for, 
for example, broadcasting. 

As a method for avoiding this problem, there is a 
method of reducing the multiplexing rate. However, if 
such a method is used, a high bit rate cannot be 
achieved. There is also a method of using an apparatus 
capable of performing multiplexing scheduling process- 
ing and bit stream multiplexing processing at a high 
speed. The entire apparatus for the processing of this 
method, however, is disadvantageously expensive. Fur- 
ther, there is a method of providing a buffer having a 
large capacity in a stage following the scheduler and bit 
stream multiplexer 122, sufficiently accumulating the 
multiplexed stream in the buffer and thereafter output- 
ting the multiplexed stream. The entire apparatus for the 
processing of this method, however, is disadvanta- 
geously expensive and large in size. 

In a case where images formed by a video camera, 
for example, are multiplexed by the above-described 
multiplexing apparatus and recorded by a recording 
apparatus, if a waiting time of the recording apparatus 
occurs, it propagates as a processing delay from bit 



stream multiplexing processing to multiplexing schedul- 
ing processing. In such a situation, processing of the 
images successively output from the video camera can- 
not be performed suitably. To prevent such waiting time 
5 propagation, a large-capacity buffer is also required, 
result in increases in the cost and size of the entire 
apparatus. 

Fig. 12 shows the configuration of an example of a 
multiplexing apparatus which performs multiplexing 

w scheduling processing and bit stream multiplexing 
processing separately from each other. 

The sizes and decoding (display) times of access 
units constituting elementary streams output from 
encoders 121 -i to 121 N are supplied as access unit 

is information to a scheduler 131 , in which data to be next 
multiplexed and the size of the data, i.e., schedule infor- 
mation, are calculated on the basis of access informa- 
tion. 

While, in the arrangement shown in Fig. 12. access 

20 unit information is obtained from the encoders 121 ^ to 
121 N outputting elementary streams, processing for 
extracting access unit information from elementary 
streams may be performed as a preprocessing before 
multiplexing scheduling processing if the encoders 12^ 

25 to 1 2 1 N cannot output access unit information . 

It is not necessary to limit the information output as 
access unit information by the encoders 121 1 to 121 N to 
the sizes and decoding times of access units. Informa- 
tion from which the sizes and decoding times of access 

30 units can be obtained may suffice. For example, with 
respect to image data encoded in accordance with the 
IS01 3818-2 standard, relative values of decoding times 
can be obtained if the frame rates of elementary 
streams of the image data and 

35 repeat_first_fieldy|3icture_coding_type of access units 
are known. 

The scheduler 131 sets, for example, a packet unit 
or the like as one schedule unit, calculates schedule 
information and additional data on the basis of access 

40 information, and commands (outputs a multiplexing 
command to) a bit stream multiplexer 132 to perform 
multiplexing by outputting these information and data to 
the bit stream multiplexer 132. 

The bit stream multiplexer 132 is supplied with ele- 

45 mentary streams from the encoders 121 1 to 121 N , and 
time-division-multiplexes the elementary streams and 
additional data from the scheduler 1 31 by a multiplexing 
command from the scheduler 131 and in accordance 
with schedule information from the scheduler 131. 

so After finishing the bit stream multiplexing process- 
ing, the bit stream multiplexer 132 outputs a multiplexing 
end notice to the scheduler 131 to inform the same of 
the end of the processing, and warts for the next multi- 
plexing command from the scheduler 131 . On the other 

55 hand, when the scheduler 131 receives a multiplexing 
end notice, it outputs a multiplexing command together 
with the next schedule information and additional data if 
the calculation of the next schedule information and 
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additional data has been completed. 

Successively, the scheduler 131 and the bit stream 
multiplexer 132 repeat the same processings. 

As described above, the scheduler 131 and the bit 
stream multiplexer 132 respectively perform multiplex- 
ing scheduling processing and bit stream multiplexing 
processing separately from each other. In such a case, 
while the bit stream multiplexer 132 is performing bit 
stream multiplexing processing, the scheduler 131 can 
start the next multiplexing scheduling processing. Thus, 
the influence of a delay of bit stream multiplexing 
processing upon the scheduler 131 is reduced in com- 
parison with the case shown in Fig. 10. It is also possi- 
ble to speed up the processing as a whole. 

However, the scheduler 131 cannot transmit the 
next schedule information and additional data to the bit 
stream multiplexer 132 unless it receives a multiplexing 
end notice from the bit stream multiplexer 132. There- 
fore, if, for example, as shown in Fig. 13A, a long wait of 
outputting of the multiplexed bit stream by the bit stream 
multiplexer 132 occurs, the delay of the outputting prop- 
agates to multiplexing scheduling processing of the 
scheduler 131. That is, in such a case, the scheduler 
131 cannot start the next multiplexing scheduling 
processing before it receives a multiplexing end notice. 
Fig. 13A shows a state in which bit stream multiplexing 
processing M(n + 1) of the (n + 1)th packet is delayed 
and multiplexing scheduling processing S(n + 3) of the 
(n + 3)th packet cannot be started. 

If, for example, as shown in Fig. 13B, multiplexing 
scheduling processing is delayed for a long time by an 
interrupt in multiplexing scheduling processing, bit 
stream multiplexing processing cannot be started, also 
resulting in a delay of the processing. Fig. 13B shows a 
state in which multiplexing scheduling processing S(n + 
1) of the (n + 1)th packet is delayed and bit stream mul- 
tiplexing processing M(n + 1 ) of the (n + 1 )th packet can- 
not be started. 

That is, in the multiplexing apparatus shown in Fig. 
12, since multiplexing scheduling processing is per- 
formed with a one-cycle precedence, a delay corre- 
sponding to one processing cycle can be absorbed. 
However, there is no means for coping with a longer 
delay, and there is the same problem as that relating to 
Fig. 10. 

SUMMARY OF THE INVENTION 

In view of the above-described circumstances, an 
object of the present invention is to provide a multiplex- 
ing apparatus and method, a transmitting apparatus 
and method, and a recording medium arranged to mini- 
mize the propagation of a delay of one of multiplexing 
scheduling processing and bit stream multiplexing 
processing to the other of these processings. 

To achieve the above-described object, according 
to one aspect of the present invention, there is provided 
a multiplexing apparatus which time-division -multi- 



plexes at least one digital signal bit stream and addi- 
tional information added to the bit stream, the apparatus 
comprising schedule information calculation means for 
calculating schedule information on a schedule of time 

5 division multiplexing on the basis of access unit informa- 
tion on an access unit prescribed in the bit stream, 
schedule information storage means for storing the 
schedule information output by the schedule informa- 
tion calculation means, additional information calcula- 

io tion means for calculating the additional information, 
additional information storage means for storing the 
additional information output by the additional informa- 
tion calculation means, and multiplexing means for 
time-division-multiplexing the bit stream and the addi- 

is tional information on the basis of the schedule informa- 
tion stored by the schedule information storage means. 

According to another aspect of the present inven- 
tion, there is provided a multiplexing method of time- 
division-multiplexing at least one digital signal bit 

20 stream and additional information added to the bit 
stream, the method comprising the steps of calculating 
schedule information on a schedule of time division mul- 
tiplexing on the basis of access unit information on an 
access unit prescribed in the bit stream, and calculating 

25 the additional information, storing the schedule informa- 
tion and the additional information, and time-division- 
multiplexing the bit stream and the additional informa- 
tion on the basis of the schedule information. 

According to still another aspect of the present 

30 invention, there are provided a transmitting apparatus, a 
transmitting method and a recording medium arranged 
to transmit or record a multiplexed stream formed by 
time-division-multiplexing at least one digital signal bit 
stream and additional information be added to the bit 

35 stream. The multiplexed stream transmitted by the 
transmitted by these apparatus and method or recorded 
on the recording medium is obtained by calculating 
schedule information on a schedule of time division mul- 
tiplexing on the basis of access unit information on an 

40 access unit prescribed in the bit stream, by calculating 
the additional information, by storing the schedule infor- 
mation and the additional information, and by time-divi- 
sion-multiplexing the bit stream and the additional 
information on the basis of the schedule information. 

45 In the above-described multiplexing apparatus, the 
schedule information calculation means calculates 
schedule information on a schedule of time division mul- 
tiplexing on the basis of access unit information on an 
access unit prescribed in a bit stream, and the schedule 

so information storage means stores the schedule informa- 
tion. The additional information calculation means cal- 
culates additional information, and the additional 
information storage means stores the additional infor- 
mation. The multiplexing means time-division-murti- 

55 plexes the bit stream and the additional information on 
the basis of the schedule information stored by the 
schedule information storage means. 

In the above-described multiplexing method, 
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schedule information on a schedule of time division mul- 
tiplexing is calculated on the basis of access unit infor- 
mation on an access unit prescribed in a bit stream, 
additional information is calculated, the schedule infor- 
mation and the addtional information are stored, and 
the bit stream and the additional information are time- 
division-multiplexed on the basis of the schedule infor- 
mation. 

In the above-described transmitting apparatus, 
transmitting method, the recording medium, a multi- 
plexed stream is obtained by calculating schedule infor- 
mation on a schedule of time division multiplexing on 
the basis of access unit information on an access unit 
prescribed in a bit stream, by calculating additional 
information, by storing the schedule information and the 
additional information, and by time-division-multiplexing 
the bit stream and the additional information on the 
basis of the schedule information. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing the configuration 
of a first embodiment of a multiplexing apparatus to 
which the present invention has been applied; 
Fig. 2 is a timing chart showing timing of process- 
ings performed by scheduler 1 and bit stream multi- 
plexer 4 shown in Fig. 1; 

Fig. 3 is a flowchart for explaining the processing 

performed by scheduler 1 shown in Fig. 1; 

Fig. 4 is a flowchart for explaining the processing 

performed by bit stream multiplexer 4 shown in Fig. 

1; 

Fig. 5 is a block diagram showing the configuration 
of a second embodiment of the multiplexing appa- 
ratus to which the present invention has been 
applied; 

Fig. 6 is a block diagram showing the configuration 
of an embodiment of a personal computer to which 
the present invention has been applied; 
Fig. 7 is a block diagram showing the configuration 
of an example of a conventional multiplex- 
ing/demultiplexing apparatus; 
Fig. 8 is a block diagram showing the configuration 
of STD; 

Figs. 9A and 9B are diagrams of the formats of mul- 
tiplexed streams, Fig. 9A showing the structure of a 
transport stream; Fig. 9B showing the structure of a 
program stream; 

Fig. 10 is a blockdiagram showing the configuration 
of an example of a conventional multiplexing appa- 
ratus; 

Figs. 11 A and 11 B are timing charts of processing 
in the multiplexing apparatus shown in Fig. 10. Fig. 
1 1 A showing an example of influence of an output 
wait causing a delay of scheduling. Fig. 1 1 B show- 
ing an example of influence of a delay of scheduling 
causing a delay of bit stream multiplexing output; 
Fig. 1 2 is a blockdiagram showing the configuration 



of another example of the conventional multiplexing 
apparatus; and 

Figs. 13A and 13B are timing charts of processings 
in the multiplexing apparatus shown in Fig. 12, Fig. 
5 13A showing an example of influence of an output 
wait causing a delay of scheduling. Fig. 13B show- 
ing an example of influence of a delay of scheduling 
causing a delay of bit stream multiplexing output. 

10 DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

Fig. 1 shows the configuration of an embodiment of 
a multiplexing apparatus to which the present invention 

is has been applied. 

Video data, audio data and other data are input to 
encoders 121 j to 121 N , each of which compresses and 
encodes the data in accordance with a predetermined 
standard, e.g., IS013818-2, IS01 1172-2, IS013813-3, 

20 or IS01 1 1 72-3, and thereafter outputs the compressed 
and encoded data as an elementary stream. Also, the 
encoders 12^ to 121 N are arranged to obtain and out- 
put access unit information on access units constituting 
elementary streams which they output. 

25 Access unit information and elementary streams 
which are output by the encoders 12^ to 121 N are 
respectively supplied to a scheduler 1 (schedule infor- 
mation calculation means) (additional information calcu- 
lation means) and a bit stream multiplexer 4 

30 (multiplexing means). 

As mentioned above, if the encoders 121 1 to 121 N 
are incapable of calculating access unit information, the 
arrangement may be such that preprocessing for 
extracting access unit information from elementary 

35 streams is performed and access unit information 
obtained by the preprocessing is input to the scheduler 
1 . Also, it is not necessary to limit the information output 
as access unit information by the encoders 12^ to 
121 N to the sizes and decoding times of access units. 

40 Information from which the sizes and decoding times of 
access units can be obtained may suffice. 

The scheduler 1 is arranged to perform multiplexing 
scheduling processing for calculating schedule informa- 
tion and necessary additional data (additional informa- 

45 tion in a broad sense) on the basis of access unit 
information from the encoders 12^ to 121 N . Schedule 
information and additional data obtained by the sched- 
uler 1 are respectively supplied to a schedule informa- 
tion intermediate buffer 2 (schedule information storage 

so means) and an additional data intermediate buffer 3 
(additional information storage means). 

The schedule information intermediate buffer 2 is 
arranged to store schedule information supplied from 
the scheduler 1. The additional data intermediate buffer 

55 3 is arranged to store additional data supplied from the 
scheduler 1. 

The schedule information intermediate buffer 2 or 
the additional data intermediate buffer 3 has a storage 
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capacity such as to be able to store schedule informa- 
tion or additional data output from the scheduler 1 a cer- 
tain number of times. In this embodiment each of the 
schedule information intermediate buffer 2 and the addi- 
tional data intermediate buffer 3 is designed so as to 
operate as a so-called ring buffer such that input data is 
successively stored and, after the space area has been 
reduced to zero, new data is overwritten in the area 
where the oldest data is stored. 

The bit stream multiplexer 4 is arranged to read out 
schedule information from the schedule information 
intermediate buffer 2, and necessary additional data 
from the additional data intermediate buffer 3, to multi- 
plex elementary streams from the encoders 12^ to 
121 N and the necessary additional data in accordance 
with the schedule information, and outputs the result of 
the multiplexing as a multiplexed stream. 

In the thus-arranged multiplexing apparatus, the 
encoders 121 1 to 121 n encode input video data, audio 
data and other data input to them and output the 
encoded data in the form of elementary streams. The 
encoders 121 1 to 121 N also obtain and output access 
unit information on access units constituting the ele- 
mentary streams that they output. 

The access unit information and the elementary 
streams output by the encoders 12^ to 121 N are 
respectively supplied to the scheduler 1 and the bit 
stream multiplexer 4. 

When the scheduler 1 receives the access unit 
information from the encoders 121! to 121 N , it calcu- 
lates schedule information on the basis of the access 
unit information, and supplies the schedule information 
to the schedule information intermediate buffer 2 to 
store it. If some additional data is to be multiplexed 
when bit stream multiplexing processing is performed in 
accordance with the calculated schedule information, 
the scheduler 1 also calculates (forms) the correspond- 
ing additional data and supplies the calculated addi- 
tional data to the additional data intermediate buffer 3 to 
store it. 

The scheduler 1 performs multiplexing scheduling 
as described above each time it receives access unit 
information. 

On the other hand, the bit stream multiplexer 4 
reads out schedule information from the schedule infor- 
mation intermediate buffer 2, selects the data designed 
by the schedule information by the size also selected by 
the schedule information, and outputs the selected data 
as a multiplexed stream. That is, the bit stream multi- 
plexer 4 selects the data corresponding to the schedule 
information by the size corresponding to the schedule 
information from the elementary streams from the 
encoders 1 21 ^ to 1 2 1 N and the additional data stored in 
the additional data intermediate buffer 3, and outputs 
the selected data as a multiplexed stream. 

This multiplexed stream is transmitted through a 
transmission medium 6 formed of, for example, the 
Internet, a CATV network, a satellite channel, a ground 



channel or the like, or is recorded on a recording 
medium 7 formed of a magneto-optical disk, a magnetic 
tape, a phase change rewritable disk, an optical disk or 
the like. 

5 Thereafter, the bit stream multiplexer 4 reads out 
the next stored schedule information from the schedule 
information intermediate buffer 2 and successively 
repeats the bit stream multiplexing processing. 

Consequently, the scheduler 1 can perform multi- 

10 plexing scheduling processing independently of the 
progress of bit stream multiplexing processing in the bit 
stream multiplexer 4 as long as each of the schedule 
information intermediate buffer 2 and the additional data 
intermediate buffer 3 (the combination of which herein- 

75 after referred to as an intermediate buffer as occasion 
demands) is not in a buffer-full state (a state of having 
no space area and having the oldest data not yet used 
by the bit stream multiplexer 4), as shown in Fig. 2. That 
is, even if bit stream multiplexing processing is delayed 

20 by occurrence of a wait of outputting of the multiplexed 
stream by the bit stream multiplexer 4, the scheduler 1 
can advance multiplexing scheduling processing with- 
out being influenced by the delay as long as the inter- 
mediate buffer is not full. 

25 The scheduler 1 is free from a need to perform con- 
trol for starting the next multiplexing scheduling 
processing after receiving a multiplexing end notice as 
described above with reference to Fig. 12, and is also 
free from a need to perform control for outputting a mul- 

30 tiplexing command each time multiplexing scheduling 
processing is finished as described above with refer- 
ence to Fig. 1 2. As a result, the processing in the sched- 
uler 1 can be simplified. 

On the other hand, the bit stream multiplexer 4 can 

35 perform bit stream multiplexing processing independ- 
ently of the progress of multiplexing scheduling 
processing in the scheduler 1 as long as the intermedi- 
ate buffer is not empty (the intermediate buffer is a sate 
of having some stored data not yet used in bit stream 

40 multiplexing processing), as shown in Fig. 2. That is, 
even if multiplexing scheduling processing is delayed by 
occurrence of an interrupt in the scheduler 1, the bit 
stream multiplexer 4 can advance bit stream multiplex- 
ing processing without being influenced by the delay as 

45 long as the intermediate buffer is not empty. 

The bit stream multiplexer 4 is free from a need to 
perform control for starting bit stream multiplexing 
processing after receiving a multiplexing command as 
described above with reference to Fig. 12, and is also 

so free from a need to perform control for outputting a mul- 
tiplexing end notice each time bit stream multiplexing 
processing is finished as described above with refer- 
ence to Fig. 12. As a result, the processing in the bit 
stream multiplexer 4 can be simplified. 

55 It is not always necessary for the scheduler 1 to 
form fixed data in additional data, e.g., stuffing bits or a 
start code in the additional data. For example, such data 
may be stored in a fixed data memory 5 provided as 
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indicated by the broken line in Fig. 1. In such a case, 
fixed data in additional data may be read out from the 
fixed data memory 5 to be supplied to the bit stream 
multiplexer 4, and there is no need to form the fixed data 
in the scheduler 1. The processing load on the sched- 
uler 1 can be reduced in this manner. 

The processing performed by the scheduler 1 
shown in Fig. 1 will be described in more detail with ref- 
erence to the flowchart of Fig. 3. 

In step S1, in the scheduler 1, a determination is 
first made as to whether access unit information neces- 
sary for multiplexing scheduling processing ready to 
start has already been received from the encoders 1 21 1 
to 121 N . If it is determined that necessary information 
has not been received, step S1 is repeated. If it is deter- 
mined in step S1 that necessary access unit information 
has been received, the process advances to step S2 to 
calculate schedule information with respect to a sched- 
ule unit corresponding to, for example, a packet unit, 
which is a current object of multiplexing scheduling 
processing, on the basis of the access unit information. 

That is, if multiplexing of data into a multiplexed 
stream in accordance with, for example, IS013818-1 
and IS011172-1 is performed, schedule information is 
calculated so as to avoid an underflow and an overflow 
in the STD buffer supposed to exist at a receiving termi- 
nal and to be adapted to elementary streams. 

After calculating schedule information, the process 
advances to step S3 to make a determination as to 
whether the schedule information can be output to the 
schedule information intermediate buffer 2. If it is deter- 
mined in step S3 that the schedule information cannot 
be output to the schedule information intermediate 
buffer 2, that is, the schedule information intermediate 
buffer 2 is full, step S3 is repeated. 

On the other hand, if it is determined in step S3 that 
the schedule information can be output to the schedule 
information intermediate buffer 2, that is, the schedule 
information intermediate buffer 2 is not full, the process 
advances to step S4 to output the schedule information 
obtained in step S2 to the schedule information interme- 
diate buffer 2. The schedule information is thereby 
stored in the schedule information intermediate buffer 2. 

If the schedule information intermediate buffer 2 is 
full, it is necessary for the scheduler 1 to stop multiplex- 
ing scheduling processing until the buffer full state is 
canceled. In this case, however, the suspension of mul- 
tiplexing scheduling processing does not influence bit 
stream multiplexing processing performed by the bit 
stream multiplexer 4 since unused schedule information 
is stored in the schedule information intermediate buffer 
2. In this situation, since decision by the scheduler 1 as 
to whether multiplexing scheduling processing should 
be stopped depends upon whether the schedule infor- 
mation intermediate buffer 2 is full or not, it is not neces- 
sary for the scheduler 1 to receive a multiplexing end 
notice from the bit stream multiplexer 4 as in the con- 
ventional art. 



After outputting the schedule information, the proc- 
ess advances to step S5 to make a determination as to 
whether there is a need for additional data when bit 
stream multiplexing is performed in accordance with the 
s schedule information, ft it determined in step S5 that no 
additional data is required, the process returns to step 
S1 to perform the next multiplexing scheduling process- 
ing. 

If it determined in step S5 that additional data is 
w required, the process advances to step S6 to form nec- 
essary data. 

That is, if multiplexing of data into a multiplexed 
stream in accordance with, for example, IS013818-1 
and IS01 1172-1 is performed, then additional data, 

15 e.g., a pack header, a packet header, or data other than 
elementary streams is formed when it is to be output as 
a multiplexed stream. 

As described above, fixed data in additional data 
can be stored in the fixed data memory 5 and can be 

20 . read out of the memory when required. However, data 
which changes with time, e.g., a clock reference needs 
to be formed by the scheduler 1 when required. 

After forming additional data, the process advances 
to step S7 to make a determination as to whether the 

25 additional data can be output to the additional data 
intermediate buffer 3. If it is determined in step S7 that 
the additional data cannot be output to the additional 
data intermediate buffer 3, that is, the additional data 
intermediate buffer 3 is full, step S7 is repeated. 

30 On the other hand, if it is determined in step S7 that 
the additional data can be output to the additional data 
intermediate buffer 3, that is, the additional data inter- 
mediate buffer 3 is not full, the process advances to step 
S8 to output the additional data obtained in step S6 to 

35 the additional data intermediate buffer 3. The additional 
data is thereby stored in the additional data intermedi- 
ate buffer 3. 

If the additional data intermediate buffer 3 is full, it 
is necessary for the scheduler 1 to stop multiplexing 

40 scheduling processing until the buffer full state is can- 
celed. However, the suspension of multiplexing schedul- 
ing processing does not influence bit stream 
multiplexing processing performed by the bit stream 
multiplexer 4 since unused additional data is stored in 

45 the additional data intermediate buffer 3. In this situa- 
tion, since decision by the scheduler 1 as to whether 
multiplexing scheduling processing should be stopped 
depends upon whether the additional data intermediate 
buffer 3 is full or not, it is not necessary for the scheduler 

so 1 to receive a multiplexing end notice from the bit 
stream multiplexer 4 as in the conventional process. 

After outputting the additional data to the additional 
data intermediate buffer 3, the process returns to step 
S1 to perform the next multiplexing scheduling process- 

55 ing. 

In the process shown in Fig. 3, additional data is 
output after schedule information has been output. 
However, schedule information and additional informa- 
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tion may be output in the reverse order or may be output 
simultaneously with each other. 

The operation of the bit stream multiplexer 4 shown 
in Fig. 1 will be described in more detail with reference 
to the flowchart of Fig. 4. 

In step S1 1 , in the bit stream multiplexer 4, a deter- 
mination is first made as to whether some unused 
(unread) schedule information is stored in the schedule 
intermediate buffer 2. 

If it is determined in step S11 that no unused 
schedule information is stored in the schedule interme- 
diate buffer 2, that is, the schedule intermediate buffer 2 
is empty, step S1 1 is repeated. If it is determined in step 
S1 1 that some unused schedule information is stored in 
the schedule intermediate buffer 2, the process 
advances to step S12 to read out the oldest of the 
unused schedule information stored in the schedule 
intermediate buffer 2. 

If the schedule intermediate buffer 2 is empty, it is 
necessary for the bit stream multiplexer 4 to stop bit 
stream multiplexing processing until the empty state is 
canceled. However, the suspension of bit stream multi- 
plexing processing does not influence multiplexing 
scheduling processing performed by the scheduler 1 
since the schedule intermediate buffer 2 is able to store 
new schedule information. In this situation, since deci- 
sion by the bit stream multiplexer 4 as to whether multi- 
plexing scheduling processing should be stopped 
depends upon whether the schedule intermediate buffer 
2 is empty or not, it is not necessary for the bit stream 
multiplexer 4 to wait for reception of a multiplexing com- 
mand from the scheduler 1 as in the conventional art. 

After reading-out of schedule information from the 
schedule information intermediate buffer 2, the process 
advances to step S13 to make a determination as to 
whether the data designated by the schedule informa- 
tion can be selected (read out). Step S13 is repeated if 
it is determined in step 13 that the data designated by 
the schedule information read out in step 12 cannot be 
selected, that is, if the data designated by the schedule 
information is an elementary stream and if the elemen- 
tary stream has not been input to the bit stream multi- 
plexer 4, or if the data designated by the schedule 
information is additional data and if the additional data 
has not been input to the additional data intermediate 
buffer 3. 

If it is determined in step 13 that the data desig- 
nated by the schedule information read out in step 12 
can be selected, the process advances to step S14 to 
select the data and to make a determination as to 
whether the selected data can be output as a multi- 
plexed stream. 

Step S14 is repeated if it is determined in step S14 
that the data cannot be output as a multiplexed stream, 
that is, if, for example, in the case of transmitting the 
multiplexed stream through transmission medium 6, it is 
necessary to delay the multiplexed stream outputting 
time for matching to the transmission rate, or if, in the 



case of recording the multiplexed stream on recording 
medium 7, a recording apparatus (not shown) for per- 
forming this recording is in a waiting state. 

If it is determined in step S14 that the data can be 

5 output as a multiplexed stream, the process advances 
to step S15 to output the data as a multiplexed stream. 
That is, if the data designated by the schedule informa- 
tion read out in step S12 is an elementary stream, the 
elementary stream is output as a multiplexed stream. If 

10 the data designated by the schedule information is addi- 
tional data, the additional data is read out from the addi- 
tional data intermediate buffer 3 (or fixed data memory 
5) to be output as a multiplexed stream. 

The process then advances to step S16 to make a 

is determination as to whether the data has been output 
by the size designated by the schedule information read 
out in step S12. If it is determined in step S16 that the 
data has not been output by the size designated by the 
schedule information, the process returns to step S13. If 

20 it is determined in step S1 6 that the data has been out- 
put by the size designated by the schedule information, 
the process returns to step S1 1 to perform the next bit 
stream multiplexing processing. 

Fig. 5 shows the configuration of another embodi- 
es ment of the multiplexing apparatus to which the present 
invention has been applied. In Fig. 5, sections corre- 
sponding to those shown in Fig. 1 are indicated by the 
same reference characters. The same description for 
the corresponding sections will not be repeated. That is, 

30 this multiplexing apparatus is arranged in the same 
manner as the multiplexing apparatus shown in Fig. 1 
except that a common intermediate buffer 1 1 (storage 
means) is provided in place of the schedule information 
intermediate buffer 2 and the additional data intermedi- 

35 ate buffer 3. 

The common intermediate buffer 1 1 is arranged in 
the same manner as the schedule information interme- 
diate buffer 2 and the additional data intermediate buffer 
3 shown in Fig. 1 to store each of schedule information 

40 and additional data output by the scheduler 1 . That is, in 
the embodiment shown in Fig. 5, schedule information 
and additional data is stored in one common intermedi- 
ate buffer 1 1 instead of being stored separately from 
each other. 

45 In the embodiment shown in Fig. 5, if there is nec- 
essary additional data, the scheduler 1 outputs the 
additional data after outputting schedule information. 
Accordingly, in the common intermediate buffer 11, 
schedule information and necessary additional data are 

so stored in the same order as that in which they are output 
by the scheduler 1. 

On the other hand, the bit stream multiplexer 4 
reads out schedule information and necessary addi- 
tional data from the common intermediate buffer 1 1 in 

55 the same manner as in the case of reading out schedule 
information or necessary additional data from the 
schedule information intermediate buffer 2 or the addi- 
tional data intermediate buffer 3, and performs bit 
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stream multiplexing processing. 

Schedule information and necessary additional 
data output by the scheduler 1 are used in the bit stream 
multipler 4 in the same order as that in which they are 
output by the scheduler 1. Therefore, there is no prob- 
lem even if the schedule information and necessary 
additional data are stored in one common intermediate 
buffer 1 1 instead of being stored separately from each 
other. As a result, in this embodiment, the configuration 
of the multiplexing apparatus can be simplified. 

As described above, schedule information and 
additional data output by the scheduler 1 are stored in 
such a manner as to prevent a delay of one of multiplex- 
ing scheduling processing in the scheduler 1 and bit 
stream multiplexing processing in the bit stream multi- 
plexer 4 from influencing the other of these processings. 

As a result, the need for using an apparatus having 
a reduced multiplexing rate or an apparatus capable of 
performing processing at a high speed is eliminated. 
Also, there is no need for providing a large-capacity 
buffer for storing a multiplexed stream. Moreover, it is 
not necessary to transmit and receive a multiplexing 
command and a multiplexing end notice as in the con- 
ventional art. 

The above-described multiplexing apparatus can 
be applied to, for example, a personal computer which 
performs so-called desk top editing such that images 
and sounds recorded by a video camera are edited and 
are recorded after being compressed and encoded. 

Fig. 6 shows the configuration of an embodiment of 
such a personal computer 

A microprocessor 21 performs predetermined 
processings such as image and sound editing of an 
input to it. A main memory 22 stores a program exe- 
cuted by the microprocessor 21 and data necessary for 
the operation of the microprocessor 21. A frame mem- 
ory 23 is formed of a dynamic random access memory 
(DRAM) or the like and stores image data at the time of 
editing of a one-frame image or the like. A bus bridge 24 
controls data exchange between an internal bus and a 
peripheral component interconnect (PCI) local bus. 

The above-described microprocessor 21, main 
memory 22, frame memory 23 and bus bridge 24 are 
connected to each other via the internal bus. Other 
blocks are connected to each other via the PCI local 
bus. The bus bridge 24 is connected to each of the inter- 
nal bus and the PCI local bus. 

A modem 25 controls communication over a tele- 
phone circuit. For example, the modem 25 can treat, as 
objects of editing, images and sounds received from the 
Internet or the like, and can transmit edited images and 
sounds to an external terminal. 

A tuner 26 receives, for example, a television 
broadcasting signal broadcasted by using, for example, 
a ground wave, a satellite channel or a CATV network 
Images and sounds received by the tuner 26 can also 
be treated as objects of editing. 

An I/O (input/output) interface 27 outputs operation 



signals corresponding to the operation of a keyboard 28 
or a mouse 29. The keyboard 28 is operated at the time 
of inputting predetermined data or command. The 
mouse 29 is operated at the time of moving a cursor dis- 
5 played on a display 36, designating a position, or the 
like. 

An auxiliary storage interface 30 controls read and 
write of data on a compact disk recordable (CD-R) 31 , a 
hard disk (HD) 32 or the like. On the CD-R or HD 32, 

10 images and sounds provided as objects of editing, 
edited images and sounds, etc., are recorded. 

An external interface 33 is arranged to control 
inputting of images and sounds to a video camera 34, a 
video tape recorder (VTR) 35, a display 36, etc. and out- 

is putting of images and sounds from these devices. The 
video camera 34 forms images used as an object of 
editing, etc. The VTR 35 reproduces editing object 
images and sounds already recorded, and records 
edited images and sounds, etc. The display 36 displays 

20 editing object images, edited images, etc. 

On a decoder board 37, a multiplexed stream 
obtained from an encoder/multiplexing board 40 is 
demultiplexed and elementary streams thereby 
obtained are decoded. 

25 The encoder/multiplexing board 40 encodes 
images and sounds from the modem 25, tuner 26, CD- 
R 31 , HD 32, video camera 34, or VTR 35 to obtain ele- 
mentary streams, and multiplexes the elementary 
streams to form a multiplexed stream. 

30 That is, data of images (video data) is input to a 
video encoder 41, which encodes the input video data 
and outputs the encoded data as an elementary stream. 
This video data elementary stream is supplied to and 
stored in a video memory 42. The elementary stream 

35 stored in the video memory 42 is read out according to 
a need by a bit stream multiplexer 49 via the video 
encoder 41 . 

The video encoder 41 also outputs, to a scheduler 
45, access unit information on the elementary stream 
40 that it has output. 

Data of sounds (audio data) is input to an audio 
decoder 43, which encodes the input audio data and 
supplies the encoded data as an elementary stream to 
a first in first out (FIFO) memory 44, thereby storing the 
45 encoded data. The elementary stream stored in the 
FIFO memory 44 is read out by the bit stream multi- 
plexer 49 according to a need. 

The audio encoder 43 also outputs, to the sched- 
uler 45, access unit information on the elementary 
so stream that it has output. 

The scheduler 45 corresponds to the scheduler 1 
shown in Figs. 1 and 5, and is formed of an MPU (micro- 
processor) 46 and a memory 47 in this embodiment. 
The MPU 46 calculates schedule information and nec- 
55 essary additional data on the basis of access unit infor- 
mation supplied to it. and outputs the calculated 
information and data to a FIFO memory 48. 

The FIFO memory 48 corresponds to the common 
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intermediate buffer 1 1 shown in Fig. 5. The FIFO mem- 
ory 48 stores schedule information and additional data 
from the scheduler 45 (MPU 46). 

[0125J 

The bit stream multiplexer 49, corresponding to the 
bit stream multiplexer 4 shown in Figs. 1 and 5. reads 
out schedule information from the FIFO memory 48 and 
selects, in accordance with the schedule information, 
data to be output as a multiplexed stream. 

That is, when an elementary stream of video data 
or audio data is to be selected, the elementary stream is 
read out from the video memory 42 or the FIFO memory 
44 to be output as a multiplexed stream. When addi- 
tional data output by the scheduler 45 is to be selected, 
the additional data is read out from the FIFO memory 48 
to be output as a multiplexed stream. Further, when 
fixed data in additional data is to be selected, the fixed 
data is read out from a fixed data memory 50. That is, 
the fixed data memory 50 corresponds to the fixed data 
memory 5 shown in Figs. 1 and 5, and has fixed data 
stored therein. When fixed data is to be selected, the 
fixed data is read out from the fixed data memory 50 to 
be output as a multiplexed stream. 

A multiplexed stream output from the bit stream 
multiplexer 49 is supplied to a bus interface (l/F) circuit 
51 . The bus l/F circuit 51 outputs the received multi- 
plexed stream to the PCI local bus. 

The multiplexed stream output to the PCI local bus 
is supplied to, for example, the decoder board 37, on 
which the multiplexed stream is demultiplexed and 
decoded and is thereafter supplied to the VTR 35 via 
the external interface 33 to be recorded, or is supplied 
to the display 36 to be displayed. Also, the multiplexed 
stream output to the PCI local bus is supplied to, for 
example, the CD-R 31 or the HD 32 via the auxiliary 
storage interface 30 to be recorded, or is supplied to the 
microprocessor 21 via the bus bridge 24 and the inter- 
nal bus to undergo predetermined edit processing. The 
microprocessor 21 is arranged so as to be able to edit 
(by cut-editing, for example) a multiplexed stream in the 
multiplexed form as well as data obtained by demulti- 
plexing and decoding a multiplexed stream. 

Since schedule information and additional informa- 
tion output by the scheduler 45 are stored in the FIFO 
memory 48 as described above, a delay of one of the 
two processings in the scheduler 45 or the bit stream 
multiplexer 49 under a basic condition does not influ- 
ence the other of the two processings, as in the case of 
the arrangements shown in Figs. 1 and 5. 

Therefore, the MPU 46 constituting the scheduler 
45, for example, can be made to perform processings 
other than multiplexing scheduling processing, e.g., 
control of communication with the microprocessor 21, 
control of the video encoder 41 and the audio encoder 
43, and the like. 

Further, even if a transfer unit of the multiplexed 



stream transferred to the PCI local bus or the like does 
not coincide with the schedule unit corresponding to, for 
example, the packet unit, the multiplexed stream can be 
transferred as a unit most suitable for the PCI local bus 

5 or a multiplexed stream transfer destination without a 
buffer provided between the bit stream multiplexer 49 
and the bus l/F circuit 51 . 

In this embodiment, images and sounds are 
MPEG-encoded. However, the encoding method is not 

io limited to those in accordance with the MPEG stand- 
ards. The present invention can also be applied to mul- 
tiplexing of images and sounds without encoding the 
images and sounds. 

While the embodiment has been described with 

75 respect to the case of multiplexing a plurality of elemen- 
tary streams, the present invention can also be applied 
to multiplexing of one elementary stream with additional 
data. 

The present invention can be realized by means of 
20 either hardware or software. 

In the multiplexing apparatus and the multiplexing 
method of the present invention described above, 
schedule information on a schedule of time division mul- 
tiplexing is calculated on the basis of access unit infor- 
ms mation on an access unit prescribed in a bit stream, 
additional information is also calculated, and the sched- 
ule information and the additional information are 
stored. The bit stream and the additional information are 
time-division-multiplexed on the basis of the schedule 
30 information. Therefore, it is basically possible to prevent 
a delay of one of the processing for calculating schedule 
information and the processing for time-division-multi- 
plexing the bit streams and the additional information 
from influencing the other of these processings. 
35 In the transmitting method, the transmitting method 
and the recording medium in another aspect of the 
present invention, a multiplexed stream is obtained by 
calculating schedule information on a schedule of time 
division multiplexing on the basis of access unit informa- 
nt) tion on a access unit prescribed in a at least one bit 
stream, by calculating additional information, by storing 
the schedule information and the additional information, 
and by time-division-multiplexing the bit stream and the 
additional information on the basis of the schedule infor- 
ms mation. In obtaining the multiplexed stream, therefore, it 
is basically possible to prevent a delay of one of the 
processing for calculating schedule information and the 
processing for time-division-multiplexing the bit streams 
and additional information from influencing the other of 
so these processings. 

Claims 

1. A multiplexing apparatus which time-division-multi- 
55 plexes at least one digital signal bit stream and 
additional information added to the bit stream, said 
apparatus comprising: 
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schedule information calculation means for cal- 
culating schedule information on a schedule of 
time division multiplexing on the basis of 
access unit information on an access unit pre- 
scribed in the bit stream; 
schedule information storage means for storing 
the schedule information output by said sched- 
ule information calculation means; 
additional information calculation means for 
calculating the additional information; 
additional information storage means for stor- 
ing the additional information output by said 
additional information calculation means; and 
multiplexing means for time-division-multiplex- 
ing the bit stream and the additional informa- 
tion on the basis of the schedule information 
stored by said schedule information storage 
means. 

A multiplexing apparatus according to Claim 1, 
wherein said schedule information storage means 
and said additional information calculation means 
are formed by one storage means for storing infor- 
mation. 

A multiplexing method of time-division-multiplexing 
at least one digital signal bit stream and additional 
information added to the bit stream, said method 
comprising the steps of: 

calculating schedule information on a schedule 
of time division multiplexing on the basis of 
access unit information on an access unit pre- 
scribed in the bit stream, and calculating the 
additional information; 

storing the schedule information and the addi- 
tional information; and 

time-division-multiplexing the bit stream and 
the additional information on the basis of the 
schedule information. 

A transmitting apparatus which transmits a multi- 
plexed stream formed by time-division-multiplexing 
at least one digital signal bit stream and additional 
information be added to the bit stream, said appara- 
tus comprising: 
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tional information; and 

time-division-multiplexing the bit stream and 
the additional information on the basis of the 
schedule information. 

A transmitting method of transmitting a multiplexed 
stream formed by time-division-multiplexing at least 
one digital signal bit stream and additional informa- 
tion which is to be added to the bit stream, said 
method comprising the steps of: 

one of inputting the multiplexed stream and 
forming the multiplexed stream; and 
transmitting the multiplexed stream, 
the multiplexed stream being obtained by: 
calculating schedule information on a schedule 
of time division multiplexing on the basis of 
access unit information on an access unit pre- 
scribed in the bit stream, and calculating the 
additional information; 

storing the schedule information and the addi- 
tional information; and 

time-division-multiplexing the bit stream and 
the additional information on the basis of the 
schedule information. 

A recording medium on which a multiplexed stream 
formed by time-division-multiplexing at least one 
digital signal bit stream and additional information 
which is to be added to the bit stream is recorded, 

the multiplexed stream being obtained by: 
calculating schedule information on a schedule 
of time division multiplexing on the basis of 
access unit information on an access unit pre- 
scribed in the bit stream, and calculating the 
additional information; 

storing the schedule information and the addi- 
tional information; and 

time-division-multiplexing the bit stream and 
the additional information on the basis of the 
schedule information. 



45 



means for one of inputting the multiplexed 
stream and forming the multiplexed stream; 
and so 
means for transmitting the multiplexed stream, 
the multiplexed stream being obtained by: 
calculating schedule information on a schedule 
of time division multiplexing on the basis of 
access unit information on an access unit pre- ss 
scribed in the bit stream, and calculating the 
additional information; 

storing the schedule information and the addi- 
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